Media Aware Distributed Data Layout

ABSTRACT

A storage system includes a plurality of vdisks, with each vdisk containing a plurality of storage segments, and each segment providing a specific class of service (CoS) for storage. Each vdisk stores files with data and meta data distributed among its storage segments. A storage system includes a memory having multiple classes of service. The system includes an interface for storing a file as blocks of data associated with a class of service in the memory. The interface chooses the class of service for a block on a block by block basis for storage. A file system for storing a file. A file system for storing includes a plurality of vdisks. A method for storing a file.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of U.S. patent application Ser. No. 15/222,449filed Jul. 28, 2016, now U.S. Pat. No. 9,710,195, which is acontinuation of U.S. patent application Ser. No. 14/175,801 filed Feb.7, 2014, now U.S. Pat. No. 9,405,487 issued on Aug. 2, 2016, which is acontinuation of U.S. patent application Ser. No. 13/493,701 filed Jun.11, 2012, now U.S. Pat. No. 8,655,931 issued Feb. 18, 2014, which is acontinuation of U.S. patent application Ser. No. 12/218,085 filed Jul.11, 2008, now U.S. Pat. No. 8,214,404 issued Jul. 3, 2012, all of whichare incorporated by reference herein.

FIELD OF THE INVENTION

The present invention is related to the storage of a file system on aplurality of segments, each of which has a different class of service.(As used herein, references to the present invention” or “invention”relate to exemplary embodiments and not necessarily to every embodimentencompassed by the appended claims.) Additionally, the present inventionconstructs said file system from multiple sub-file systems (vdisks),with operations involving multiple vdisks providing atomicity using apersistent operations table to record partial operation state.

BACKGROUND OF THE INVENTION

This section is intended to introduce the reader to various aspects ofthe art that may be related to various aspects of the present invention.The following discussion is intended to provide information tofacilitate a better understanding of the present invention. Accordingly,it should be understood that statements in the following discussion areto be read in this light, and not as admissions of prior art.

Most file systems today lack certain features useful for supportingmixed types of storage, as well as huge amounts of storage. In addition,most file systems today have meta data bottlenecks that limit theirperformance scaling in multi-core and distributed systems. The inventionpresented here is a novel file system implementation addressing theseissues.

BRIEF SUMMARY OF THE INVENTION

This invention divides a file system into a number of pools of inodes,otherwise called vdisks, and within each inode pool, data is stored inmultiple segments, each potentially with a different class of service.Automated policies can choose the appropriate segment for differentfiles, or even different portions of a file, chosen either statically ordynamically.

Each inode pool can be served by a separate processor in amultiprocessor system, limiting the amount of inter-processorcommunication within a file system to communication between inode pools.And each inode pool can be checked for consistently independently,greatly reducing the amount of computation and IO operations required torecover from even a severe system failure or software bug resulting infile system inconsistency, since only the inode pool with the detectedinconsistency needs to be checked and repaired.

The present invention pertains to a storage system. The system comprisesa memory having a first segment for storing data having a first class ofservice, and at least a second segment for storing data having a secondclass of service which is different than the first class of service ofthe first segment. The system comprises an interface which receives afile and stores a first portion of the file in the first segment and asecond portion of the file on the second segment, and retrieves a fileby reading the first portion and the second portion from the first andsecond segment, respectively.

The present invention pertains to a storage system. The system comprisesa plurality of vdisks, with each vdisk containing a plurality of storagesegments, and each segment providing a specific class of service (CoS).Each vdisk stores files with data and meta data distributed among itsstorage segments.

The present invention pertains to a storage system. The system comprisesa memory having multiple classes of service. The system comprises aninterface for storing a file as blocks of data associated with a classof service in the memory. The interface chooses the class of service fora block on a block by block basis.

The present invention pertains to a file system for storing a file. Thesystem comprises a plurality of vdisks, with each vdisk having aplurality of inodes. Each inode of each vdisk stores data on one or moresegments, with each segment having a different class of service. Thesystem comprises a controller which stores data of a file in an inode ofa vdisk, in one or more segments of that vdisk

The present invention pertains to a file system for storing a file; thesystem comprises a plurality of vdisks, and each vdisk having aplurality of inodes. The system comprises a controller including aplurality of processors, with each processor serving one or more of thevdisks.

The present invention pertains to a file system for storing files. Thesystem comprises a plurality of vdisks, with each vdisk having aplurality of inodes, a plurality of inodes of at least one vdisk storingdata on a plurality of segments, each segment having a different classof service. The system comprises a controller which stores data of thefile in one or more segments of one vdisk.

The present invention pertains to a method for storing a file. A methodcomprises the steps of receiving the file at an interface. There is thestep of storing data of the file with a controller in one or moresegments of one vdisk of a plurality of vdisks, each vdisk having aplurality of inodes. The plurality of inodes of at least one vdisk,store data in a plurality of segments, with each segment having adifferent class of service.

The present invention pertains to a method for storing a file. Themethod comprises the steps of receiving the file at an interface. Thereis the step of storing a first portion of the file in a first segment ofa memory and a second portion of the file in a second segment of thememory. There is the step of retrieving the file by reading the firstportion and the second portion from the first and second segment,respectively.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

In the accompanying drawings, the preferred embodiment of the inventionand preferred methods of practicing the invention are illustrated inwhich:

FIG. 1 shows two servers, each with multiple drives, with each driveholding multiple chunks. Some chunks hold file system data, while othersstore RAID parity information.

FIG. 2 shows the construction of vdisk block address spaces from one ormore segments.

FIG. 3 shows an alternative construction of segments from individualdisk drives instead of from chunks of disk drives.

FIG. 4 shows the distribution of files and directories in a file systembetween multiple vdisks.

FIG. 5 shows the modular decomposition of the invention, along with thebinding of processors to vdisk module instances, and the binding ofvdisk instances to the set of drives each vdisk controls.

FIG. 6 shows the message traffic involved in a file or directory deleteoperation, in the simple case where locks are all obtained withoutconflict.

FIG. 7 shows the message traffic involved in a file or directory deleteoperation, in the case where optimistic locking fails and locks have tobe obtained again in a different order.

FIG. 8 shows the message traffic involved in a hard link operation(source locked first case).

FIG. 9 shows the message traffic involved in a hard link operation(target locked first case).

FIG. 10 shows the message traffic involved in a file create/directorycreate or symbolic link create operation.

FIG. 11 shows the message traffic involved in a rename operation.

FIG. 12 shows the fields stored in an Mode for use by the simple policymechanism.

FIG. 13 shows the modules present in a Unix operating system includingthe vdisk module instances of this invention.

FIG. 14 is a block diagram of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Referring now to the drawings wherein like reference numerals refer tosimilar or identical parts throughout the several views, and morespecifically to FIGS. 5 and 14 thereof, there is shown a storage system10. The system 10 comprises a memory 12 having a first segment 14 forstoring data having a first class of service, and at least a secondsegment 16 for storing data having a second class of service which isdifferent than the first class of service of the first segment 14. Thesystem 10 comprises an interface 18 which receives a file and stores afirst portion of the file in the first segment 14 and a second portionof the file on the second segment 16, and retrieves a file by readingthe first portion and the second portion from the first and secondsegment 14, 16, respectively.

The interface 18 can store the file within one or more stripes of datain the first segment 14 and within one or more stripes of data in thesecond segment 16. The system 10 can include a plurality of disks 20 andwherein the interface 18 stores each segment as a plurality of datachunks distributed among several disks 20. The segment can include aparity strip and the interface 18 stores the parity strip in the memory12 according to RAID techniques so if any one strip is unavailable, thedata blocks can still be retrieved from the other strips and the paritystrip. The inode describing a file includes meta data which keeps trackof the first portion and the second. The meta data is preferably aUnix-style indirect block tree.

The present invention pertains to a storage system 10. The system 10comprises a plurality of vdisks 24. Each vdisk 24 contains a pluralityof storage segments, with each segment 24 providing a specific class ofservice (CoS). Each vdisk 24 stores files with data and meta datadistributed among its storage segments.

A first portion of the file data can be stored in a first segment 14,and a second portion of the file can be stored in either the firstsegment 14 or a second segment 16. The system 10 can include a pluralityof vdisks 24, and where a vdisk 24 to hold a newly created file ordirectory is chosen from the plurality of vdisks 24 based on apredetermined mapping into the plurality of vdisks 24 in the storagesystem 10. The predetermined mapping can be a round-robin assignmentalgorithm. The predetermined mapping can choose the vdisk 24 with alargest available space, or a largest percentage of available space.

The system 10 can include a plurality of processors 26 and wherein thepredetermined mapping chooses the vdisk 24 served by a least loadedprocessor 26. Each vdisk 24 can be a random collection of directoriesand files, and at least one file is stored in at least two segments.Each vdisk 24 can be a random collection of directories and files, andat least two vdisks 24 hold at least one file. The system 10 can includean interface 18 to initiate a file system 10 consistency check on anindividual vdisk 24, triggered by an indication of an inconsistency in aspecific vdisk 24.

The present invention pertains to a storage system 10. The system 10comprises a memory 12 having multiple classes of service. The system 10comprises an interface 18 for storing a file as blocks of dataassociated with a class of service in the memory 12, with the interface18 choosing the class of service for a block on a block by block basis.

The present invention pertains to a file system 10 for storing a file.The system 10 comprises a plurality of vdisks 24, with each vdisk 24having a plurality of inodes. Each inode of each vdisk 24 stores data onone or more segments, with each segment having a different class ofservice. The system 10 comprises a controller 28 which stores data of afile in an inode of a vdisk 24, in one or more segments of that vdisk24.

The system 10 can include a plurality of processors 26, with eachprocessor 26 serving at least one of the vdisks 24. Each segment canhave a plurality of chunks. Each segment can have chunks added to itdynamically over time. Each vdisk 24 can include an inode table 30describing each file in the vdisk 24. One of the vdisks 24 is preferablya root vdisk 24. One of the inodes in the root vdisk 24 is preferably aroot inode of the file system 10. The system 10 can include at least onedirectory storing mappings of file names to inode pointers. Each inodecan have a back pointer to the directory entry pointing to the inode.

The file system 10 described in this invention is preferably a tree,with a single top-most directory, containing a mix of files anddirectories. Each other directory also contains a set of files and otherdirectories. So, the root inode is the topmost inode in the file system10 tree, and the only directory that doesn't have a parent directory.

Directories, including the root directory, are just normal files, markedwith a special file type so that they can't just be read and written byusers, but can instead only have directory operations like “create filein directory” performed on them. But, like normal files, they exist in asingle vdisk and have their data blocks stored in one or more segmentsin that vdisk.

The present invention pertains to a file system 10 for storing a file.The system 10 comprises a plurality of vdisks 24, with each vdisk 24having a plurality of inodes. The system 10 comprises a controller 28including a plurality of processors 26, with each processor 26 servingone or more of the vnodes.

The present invention pertains to a file system 10 for storing a file.The system 10 comprises a plurality of vdisks 24, with each vdisk 24having a plurality of inodes, a plurality of inodes of at least onevdisk 24 storing data on a plurality of segments, each segment having adifferent class of service. The system 10 comprises a controller 28which stores data of the file in multiple segments of one vdisk 24.

The present invention pertains to a method for storing a file. A methodcomprises the steps of receiving the file at an interface 18. There isthe step of storing data of the file with a controller 28 in multiplesegments of one vdisk 24 of a plurality of vdisks 24. Each vdisk 24stores a plurality of inodes. At least one inode of at least one vdisk24 stores data in a plurality of segments, each segment having adifferent class of service.

The present invention pertains to a method for storing a file. Themethod comprises the steps of receiving the file at an interface 18.There is the step of storing a first portion of the file in a firstsegment 14 of a memory 12 and storing a second portion of the file in asecond segment 16 of the memory 12. There is the step of retrieving thefile by reading the first portion and the second portion from the firstand second segment 14, 16, respectively.

In the operation of the invention, physical disks 20 and/or RAID arraysare divided into fixed sized chunks of storage with identical or verysimilar performance and reliability characteristics. These chunks maystore data or parity (or checksum) information. These chunks arecombined into variable sized segments, each segment providing a lineardisk block address space, as well as a meta data description of theclass of storage provided, including RAID class, average seek time, andread and write data transfer rates. Chunks may be combined into segmentsby simple concatenation, as shown by the diamond or horizontally markedsegments in FIG. 1, in which chunks are concatenated sequentially in theorder of their tag. Chunks may also be combined with RAID parityprotection, as shown by the diagonally marked or shaded segments in thesame figure, where, in these examples, every three chunks of data isstored with an additional parity chunk that stores the RAID 5 parityinformation for those three chunks. In the case of segments with paritychunks, the parity data is not included in the segment's linear addressspace, so that the linear address is comprised of the concatenation ofthe data chunks only. For example, the linear address space of theshaded segment in FIG. 1 is comprised of the ordered set of chunks {0,1, 2, 3, 4, 5}.

A chunk is the smallest part of the disk that we *assign* to one segmentor another. A block, on the other hand, is the smallest addressable partof a disk for the purposes of doing *I/O*. That is, we do disk spaceallocation in units of chunks, but we do individual reads and writes ata more granular level.

Segments are a simple block storage concept, and are combined intosparse linear address spaces called vdisks 24. Different segments withina vdisk 24 may have varying storage types. A vdisk 24 also includes aninode table 30, with each entry identified by a 64 bit inode ID,specifying the inode within the vdisk 24. Within a vdisk 24, any inode'sdata or meta data blocks may be stored on any of the segments within thevdisk 24. For example, a policy might specify that the first megabyte ofevery file is allocated from a segment with very low latency, with theremaining blocks allocated from segments stored on storage with higherlatencies.

A file system 10 comprises multiple vdisks 24. One vdisk 24 isdesignated the root vdisk 24, and a designated inode within that vdisk24 represents the root of the entire file system 10. The set of theinodes within the file system 10 is then the union of all of the inodesin all of the file system's 10 vdisks 24.

Directories store mappings from file names (represented as UTF-8strings, without loss of generality) to (vdisk 24, inode) pairs. Eachinode has a back pointer (or a set of back pointers) to the directoryentry or entries pointing to the Mode, to help in validating thedirectory entry. These back pointers also allow directory reconstructionby scanning for Modes stored in the directory, and can also be used ingenerating path names from inodes.

In FIG. 1, each small box is a chunk of storage, allocated from a driverepresented by a horizontal box. FIG. 1 contains two logical servers,possibly residing within the same computer system, the first of whichsupports three drives, and the second of which supports four drives.Each logical server provides storage for the chunks making up one or twosegments. Each segment is striped among that server's drives: server 1stores chunks from the dotted segment, and server 2 stores chunks fromthe shaded and white segments. Each segment provides storage with adifferent class of service: the dotted segment on sever 1 provides RAID0 storage; the shaded segment on sever 2 provides RAID 5 storage, andthe white segment on server 2 provides additional RAID 0 storage.

Note that RAID is implemented in this figure across chunks stored ondifferent drives, rather than across multiple drives in their entirety.This means that for RAID segments, some of the segment's chunks storeparity information instead of file system data. For example, drive 7'sleftmost shaded chunk stores parity information for chunks 0, 1 and 2for the shaded segment, and drive 6's rightmost shaded chunk storesparity information for the shaded segment's chunks 5, 4 and 3.

FIG. 2 shows the shaded, white and dotted chunks being combined intoshaded, white and dotted segments. The dotted and white segments make upthe block address space for vdisk 1, so that files whose inodes arestored on vdisk 1 can have blocks placed on either of the thesesegments, as the class of service policies for that file dictate. Thisinvention's ability to aggregate multiple types of segments in a singlevdisk 24 allows the dynamic allocation of storage with a desired CoS toportions of a file. Vdisk 2 consists of only the shaded segment. Incommon practice, when creating separate vdisks 24 for scalability andfault isolation reasons, each vdisk 24 would typically be comprised ofsets of similarly typed segments.

Each vdisk 24 provides a separate Mode space, and in this example, thetwo vdisks 24 are combined into a single file system name space. Theroot is a specially designated Mode in a designated vdisk 24, and filesare allocated out of each vdisk 24 on either a random, or apolicy-driven, basis as new files and directories are created. It is notexpected to move files between vdisks 24 frequently, or eventransparently; changing the class of service of all or part of a file isaccomplished not by changing a file's vdisk 24, but by migrating afile's individual blocks to those segments within the file's originalvdisk 24 providing the desired class of service. Overall load balancingcan be accomplished by moving the responsibility for processing entirevdisks 24 between processors 26 or systems 10.

Note that vdisk 1 is actually comprised of storage (segments) controlledby both server 1 and server 2. A file allocated on vdisk 1 could haveits blocks allocated half from the diagonally marked segment, and halffrom the shaded segment. When vdisk 1's owner needs to read or writedata on another logical server, it does so by sending a request to thatserver to perform the read or write operation on its behalf. Thisflexibility allows one to build loosely coupled servers serving a singlefile system 10.

In other words, vdisks 24 have owning processes that control the metadata operations for all segments within that vdisk 24. Each segment hasits own controlling process performing its basic read and writeoperations. In some cases, the process controlling a vdisk 24 is thesame as the process controlling all of that vdisk's 24 segments, butthis need not be the case in general.

Note that alternative mechanisms for creating segments from disk storageare also possible. For example, FIG. 3 shows segments comprised ofentire RAID groups, with each RAID group made from chunks comprised ofentire disk drives. As in the previous example, the segments are mappedinto the vdisk 24 address space sparsely, to provide room for additionalgrowth in an existing segment.

FIG. 4 shows the mapping between a file system directory structure and acollection of vdisks 24. In this example, vdisk 1 (the horizontalstripes) holds the root directory, and stores references to another setof directories, some of which are located on the same vdisk 24 and someof which are stored on vdisk 2. Each of those directories in turn storereferences to some files (in this example), again, some of which are onthe same vdisk 24 as the parent directory, and some of which are on adifferent vdisk 24. In this invention, each object within a directorymay be located on any vdisk 24 in the system 10, whether or not it isthe same vdisk 24 as holds the directory.

Meta data operations that affect a single file are typically done by adesignated vdisk 24 owner, typically updating local or remote(preferentially local) segments containing the appropriate meta data.

This architecture allows several different forms of striping. The fileswithin a directory will typically be striped among multiple vdisks 24,allowing concurrent data and meta data operations on different fileswithin even a single directory. In addition, a file's blocks can beallocated from multiple segments, each controlled by a different server,and allowing multiple processes to perform IO operations for differentportions of even a single file.

Aside from meta data operations that affect a single file, some vdisk 24operations affect multiple files, located in multiple vdisks 24. All ofthese operations, either affecting one or affecting more than one vdisk24, are described below.

The implementation of a device to implement the vdisk 24 interface 18 isnow described. The vdisk 24 interface 18 plugs into a standard Unixoperating system kernel just below the vnode layer, with a thin gluelayer mapping incoming vnode calls from the kernel into the vdisk 24operations described here. This glue layer is described in detail below,but every significant vnode operation has a corresponding vdisk 24 layeroperation.

Referring to FIG. 5, a file create followed by a write of new data tothe newly created file is described. The figure illustrates a system 10with two processors 26. Processor 1 has software ownership of drives 1and 2 (the horizontal boxes below the processor 26), containing twosegments, A and B. Segments A and B provide storage to vdisk 1, whichstores a subset of the inodes in the single file system 10 exported inthis example. Similarly, processor 2 has ownership of drives 3 and 4,which collectively store segment C. Segment C is the sole segmentproviding storage for vdisk 2, which stores the remainder of the inodesin this exported file system 10. All operations on vdisk 1, segments Aand B, and drives 1 and 2, are performed by processor 1, while alloperations on vdisk 2, segment C and drives 3 and 4 are performed byprocessor 2.

A file create request from the protocol servers (NFSv3 and CIFS serversin FIG. 13), through a vnode shim layer, and finally into the top of thevdisk layer as a file create request contains a file handle identifyingthe directory in which to create the file, and also contains the name ofthe file to be created. The create operation returns the file handle ofthe newly created file. FIG. 13 shows how file system 10 requests enterthe system 10.

The file create request begins by consulting a vdisk 24 locationservice, accessible from any processor 26, to determine the currentowner of the target vdisk 24 storing the directory in which the new fileis to be created; the target vdisk 24 can be located in a subfield ofthe incoming directory file handle. The create request is then forwardedto the processor 26 owning this vdisk 24. If it is assumed the targetdirectory resides on vdisk 1, then the file create request beginsexecution on processor 1.

The vdisk 24 module for vdisk 1 will choose the vdisk 24 to hold thefile to be created, based upon policies such as load balancing orbalancing the space used by the various vdisks 24. In this example, itis assumed the policy module chooses vdisk 2 to hold the new file. Sincethe file create operation needs to atomically create a file by changingboth the directory's vdisk 24, and the new file's vdisk 24, the createoperation begins by creating a transaction entry in the directoryvdisk's 24 (vdisk 1's) persistent operations table (POT) 22, storing theparameters of the operation tagged with a unique transaction ID. In theevent of a system failure, this information can be used to restart theoperation. The first vdisk 24 then sends a perform-object-createoperation to the second vdisk 24, with this same transaction ID. Thesecond vdisk 24 now creates a file in its inode table 30, updatingstorage in segment C. As part of this operation, vdisk 2 creates its ownPOT 22 entry in its own POT 22, tagged by the same transaction ID, andgiving the inode number of the created object. This object's file handle(derived from its inode number) is passed back in theperform-object-create's response to vdisk 1. The create operation invdisk 1, running again on processor 1, completes the new directory entryso that it stores both the new file name and the newly created file'sinode number. It then marks the POT 22 entry for this operation ascomplete. Cleanup of the POT 22 entries is described below.

As part of creating the directory entry, vdisk 1 needs to update thecontents of the new file's parent directory. It does this by updatingdisk blocks on either segment A or segment B, depending upon thedirectory's associated storage policy. Similarly, as part of allocatinga new object (inode), vdisk 2 allocates an inode by writing to segmentC.

This illustrates several key features of this invention. First, theinvention's ability to create files on an arbitrary, policy specified,vdisk 24, greatly simplifies load balancing in the system 10, since noadministrator defined volume boundaries between vdisks 24 exist. Thisdescription also illustrates how a storage policy associated with adirectory can guide the vdisk's 24 write operation to choose a storagesegment with the desired class of service, allowing a policy to specifya different class of service at as fine a level of granularity as anindividual disk block.

Next is described the writing of a block of data to the newly createdfile. The write request specifies the file to update by file handle. Inthe example, the write request will be received by an arbitraryprocessor 26, and since the file was created on vdisk 2, the requestwill be forwarded to vdisk 2's processor 26, which is processor 2. Thatprocessor 26 will call vdisk 2's write operation, which will allocate ablock from segment C (the policy module will not have much to do in thiscase, since there is only one segment to choose from), and then writethe updated data to drive 3 or drive 4, as required.

Next, the details of the various vdisk 24 operations provided areexamined.

The vdisk 24 operations are divided into three classes. The first setconsists of simple operations that affect a single file at a time. Thesecond set consists of operations that read the contents of directories,either to lookup an individual object, or to read the contents of adirectory, possibly returning file attributes simultaneously. Finally, athird set of operations consists of operations that modify directories,by creating, deleting or renaming file objects within the directory ordirectories.

Increased concurrency compared with the state of the art can be obtainedfor operations in the first set above easily by executing the operationsfor each vdisk 24 on a separate processor 26. Since these operationsrequire no shared state between different vdisks 24, the operations canexecute completely concurrently on separate vdisks 24. Similarly, thesecond set of operations either operates on a single vdisk 24 as well(the readdir operation), or naturally decomposes into two separateoperations which run first on one vdisk 24 and then on another (lookup,readdirplus), which can also run with high concurrency. Even the thirdset of operations, those that modify one or more directory entries, onlyaffect a small number of vdisks 24 (typically two, but occasionallythree or four), meaning that in a system 10 with dozens of vdisks 24,many such operations can run concurrently without creating a bottleneckon a single vdisk 24. Again, this allows considerably improved levels ofconcurrency, as compared with more centralized file systemarchitectures.

One of the key innovations of this invention is the implementation ofdirectory modifying operations as multi-stage operations, where eachstage affects a single vdisk 24, and where the overall operation and itsprogress is recorded persistently so that the operation is performedatomically, and can complete successfully even if multiple processors 26involved in the operation repeatedly restart during the operation; thisis described further below.

Details of the implementation of these operations are provided below.

The vdisk 24 interface 18 exports a set of simple operations affecting asingle file or directory, which are very straightforward to implement inthis invention. Each operation is performed on a single file, and readsor updates either the status of the file or its data blocks. Any vdisk24 implementation that supports inode structures comparable to theFreeBSD operating system's UFS file system can use the inodes stored ina file system 10 as a vdisk 24 implementation for these operations.

The vdisk 24 interface 18 includes the following single file operations:

-   -   getattr—get the attributes of a file    -   setattr—change the attributes or file length of a file    -   read—read data from a file    -   write—update the contents of a file, and update the inode change        time (POSIX file system ctime) and data modification time (POSIX        file system mtime) fields simultaneously.    -   readdir—read the contents of a directory in a standard format.

The write operation ties into the choice of segment for a file.Specifically, the implementation of a write operation will create dirtybuffers in the memory 12 cache that are tagged by the inode identifyingthe file, and the offset within the file at which the data is located. Abackground “cleaner” thread within the vdisk 24 module will examine theinode for its link to a predetermined class of service policy. Thispolicy might be, for example, that the first N megabytes of a file'sdata should be written to the segment in the vdisk 24 with the lowestread and write latency, and the remaining data should be written to thesegment with the largest available free space; this would be an exampleof a static policy, since the parameters of the policy are defined once,and then followed for multiple files. The cleaner would then allocateblocks of data for the dirty buffers from the vdisk's 24 segmentsaccording to the inode's policy and then write the dirty data to thosenewly allocated blocks.

A more dynamic policy might write the most heavily accessed small filesto a segment comprised of flash, or other low latency storage. In thiscase, some simple per-inode statistics would need to be gathered by thevdisk 24 manager so that the policies for heavily accessed files couldbe applied to the correct set of files.

Two vdisk 24 operations, lookup and readdirplus, interpret the contentsof a directory, while also returning the attributes associated with oneor more files in that directory.

The lookup operation searches the directory for an entry with a specificname, and returns the file's inode number (which, when qualified withthe vdisk's 24 ID, gives the file's file handle), along with the file'sattributes. To obtain the file attributes, the implementation of thevdisk lookup operation will send a vdisk 24 getattr call to the vdisk 24server, and return the attributes of the file along with the rest of thevnode lookup results. Making this call to the vdisk server 24 allows thelookup call to handle the case where the target file is located inanother vdisk 24.

Similarly, readdirplus returns a set of file names, along with theattributes of each of the files. Each separate file's attributes may, asin the lookup case, come from a different vdisk 24, and, as with lookup,the attributes for these files come from vdisk 24 getattr cals made tothe other vdisk(s) 24.

Neither lookup nor readdirplus make any guarantees about reading thedirectory and obtaining the file attributes atomically, so theimplementation can straightforwardly be done in two steps, first readingthe directory's contents, and then obtaining the target file'sattributes.

Directory modification operations are considerably more complex, in thatthey involve modifying multiple vdisks 24 atomically. To implement thesemulti-stage operations atomically, they are implemented as transactions,and make use of a persistent operations table (the POT 22) in each vdisk24. The POT 22 entry stores the status of each complex operation, aswell as any lock state involved in the operation. Each operation isimplemented as a persistent state machine, recording the current statein the persistent operations table (POT 22). For a given operation, eachvdisk 24 stores its own POT 22 entry for its portion of the transaction.

Each operation in the POT 22 is assigned a UUID when it first arrives atthe first vdisk 24 receiving the vdisk 24 operation; this is called theprimary POT 22 entry. The execution of the operation may require sendingrequests to other vdisks 24 storing other objects modified by this vdisk24 operation. These are called secondary requests, and their executionat a vdisk 24 may create secondary POT 22 entries in their local vdisk'spersistent operations table. These secondary POT 22 entries are taggedwith the same request UUID, and are used to ensure that sub-componentsof atomic operations are executed exactly once, even in the case ofmultiple server crashes and restarts.

Some secondary operations set locks on files, directories, or portionsof directories. Locks in this system 10 are comprised of either a lockon an entire file handle, or a lock on a specific file name within adirectory file handle. Locks on file handles conflict only if theyspecify the same file handle. Locks with names and file handles conflictonly if both components match exactly. If one lock is a whole filehandle lock and the other is a name lock, they conflict only if the filehandle components match. Independent of the above, two read locks neverconflict, even if the file handles would otherwise conflict.

Each class of directory modification operation in turn is now examined.

A vdisk delete or rmdir operation begins at the parent directory's vdisk24 server. The request creates a primary POT 22 entry with a new UUID,and the target file handle (including the target vdisk 24) is determinedby doing a lookup operation on the directory. Before doing the lookupoperation, the request establishes a lock on the <parent-dir, name>mapping, and once the local lookup operation completes, the operationsends a perform-unlink-target request to the target vdisk 24 tagged withthe POT 22 entry's UUID. The perform-unlink-target operation sets awrite lock on the target file handle; a parameter to this operationtells the server whether the vdisk 24 server should wait for the lock orshould instead fail the operation on a lock conflict, and for filedelete, the caller sets the “wait” flag only if the target file handlefollows the source directory lock in the global locking hierarchy. If,as is likely, there is no conflicting lock, the target object is locked,a secondary POT 22 entry is created with the request's UUID, the objectis destroyed, the lock on the target object is released, and theperform-unlink-target operation is marked as complete. While the objecthas been destroyed, and the operation is marked as complete, therequest, tagged by UUID, stays in the persistent operation table untilthe operation is also completed at the primary server, to ensure thatretransmissions of the perform-unlink-target operation are detected, andnot re-executed. Once the perform-unlink-target call has completed, theprimary vdisk 24 removes the file name from the source directory, anddrops all of its local locks. The primary removes the operation's POT 22entry is removed from its POT 22, and a response is sent to theoriginator of the request. In addition, the request's UUID is batched upand sent eventually to the target object's vdisk 24 as part of a batchpersistent-operation-complete request. Upon receipt of this message, thesecondary removes the operation from its persistent-operation-table aswell, since at this point, it knows that the operation will never beretransmitted (as it has been removed from the primary server). Notethat if a crash occurs before the persistent-operation-complete requesthas been processed, the secondary can iterate over all of its pendingoperations, checking with the primary vdisk 24 to see if the operationis still in progress; if the secondary finds any operations that are nolonger present in the primary's POT 22, the secondary can remove theoperation from its table, as well.

FIG. 6 shows the message flow for this example.

FIG. 6 shows the behavior of the system 10 when the target succeeds atobtaining its lock, or if the target waits for its lock. It is alsopossible that the target failed to obtain its lock on the target filehandle. In this case, the source vdisk 24 releases its locks, and triesthe operation in two phases, first locking the target, and then doingthe actual work. In this case, the primary vdisk 24 server sends aslightly different request, prepare-unlink-target to the target. Theprepare-unlink-target specifies that upon a conflict, the request shouldwait for the lock. After the primary gets the response, the source locksthe source handle, and verifies that the name still matches the lockedobject. If it doesn't, the entire operation is restarted; otherwise thesource server marks the operation as committed, and removes the entryfrom the directory, while concurrently telling the prepared target vdisk24 to remove the target object, by sending it a commit-unlink-targetoperation. Once the removal is complete, the source vdisk 24 completesthe operation and removes the operation from the persistent operationtable. FIG. 7 shows the behavior of the system 10 if the lock can't beobtained initially.

The hard link operation works fairly similarly to remove, but is simplerbecause all of the file handles are known at the start of the operation.The parameters to link include a source directory and file name, and atarget file handle that may be part of another vdisk 24. The sourcevdisk 24 determines the locking order between the target file and thesource directory. The operation begins, as usual, by creating a requestwith a new UUID on the source vdisk 24.

If the source needs to be locked first, the source vdisk 24 locks thedirectory+name, and then sends a perform-link-target operation that doesthe locking for the link target, and updates the attributes as well(primarily the link count and ctime fields). Once the source receivesthe response, its local entry transitions to “complete” state andbatches up a cleanup operation to the target vdisk 24, upon receipt ofwhich the target can remove the operation from its persistent operationtable and release all of its locks. FIG. 8 illustrates this messageflow.

If the target needs to be locked first, the source then sends it aprepare-link-target request, which locks the target file. The sourcethen locks the source directory+name, ensures the entry doesn't exist,and creates it. Finally, it sends the target a commit-link-targetrequest. The target executes the request, and keeps it in its POT 22until the source contacts it, as part of a piggy-backed operation,indicating that the operation is complete and the request has beendeleted. At this point, the target can remove the operation from itspersistent operation table. FIG. 9 illustrates the message flow in thiscase.

Create and mkdir function similarly to each other. Because bothoperations create new entities, rather than dealing with existingentities that might be active, they are simpler than remove/rmdir toimplement.

The operation begins at the directory's vdisk 24 (the primary vdisk 24),where the operation locks the file handle+name, and adds a new createrequest to the persistent operations table.

The target vdisk 24 is chosen either via a policy associated with theparent directory in which the object is being created, or via a globalpolicy associated with the entire file system 10. No matter what thepolicy's source, the policy can select a vdisk 24 on which to create thenew object based on the load on the vdisk 24, the operations/secondcapacity of the vdisk 24, the space available on the segments comprisingthe vdisk 24, or any other function of the state and configuration ofthe system's vdisks 24.

The target object's vdisk 24 (the secondary vdisk 24) then receives aperform-object-create operation, telling it the type of object to create(empty directory, symbolic link, or file). The object is created and theresponse is returned to the primary vdisk 24, along with the identity ofthe newly created object. The primary vdisk 24 then creates thedirectory entry with the new object name and file handle, and marks theoperation as complete. The vdisk 24 then batches up the completionnotification to the secondary vdisk 24, which removes the operation fromits persistent operations table.

FIG. 10 illustrates the message flow.

Since these operations actually create new objects, one of the mostimportant functions they perform is choosing a vdisk 24 to hold the newfile, directory, or symbolic link. There are many potential algorithmsto follow here, and the specific algorithm can be chosen either as apolicy associated with the parent directory, or by a more global policy,perhaps pertaining to the entire global file system 10.

One vdisk 24 choice function might be to create a new object on thevdisk 24 with the most free space, or the greatest percentage of freespace; this would automatically keep space utilization balanced. Anothervdisk 24 choice function might be to create a new object on the vdisk 24with the lowest operation rate, i.e. on the vdisk 24 with the most sparecapacity in operations/second. Obviously hybrid scoring approaches thatcombine a “free space” store and a “available operations/second” scorecould also be used. For example, the choice function could compute alinear combination of the two scores, and the invention could thencreate new objects on the vdisk 24 with the lowest hybrid score. Otherhybrid functions might be include quadratic functions over the relevantscores, as well.

Of course, a very simple vdisk 24 choice function might be a simpleround-robin algorithm, where vdisks 24 are chosen in a repeated,circular pattern, but it is unlikely that such an approach would beoptimal under any useful metric.

The rename operation can involve up to four separate objects: a sourcedirectory, a target directory, a source file or directory, and a targetfile or directory. All four, in the worst case, may be located onseparate vdisks 24. Semantically, the rename operation removes thedirectory entry in the source directory, and creates a new directoryentry to the same object in the target directory. If an entry alreadyexists at the target directory, it is first destroyed.

The operation begins at the source directory's vdisk 24. If the sourceand target directories are identical, a simplified operation can beperformed. The source (and thus target) directory is simply locked andthe contents of the directory are updated directly. There are two cases,depending upon whether the target of the rename exists or not. If thetarget does not exist, then nothing changes except the directory storingthe file/directory names. In this case, there is no message flow toillustrate, because all work is done by the source/target directory'svdisk 24 server.

If the target file or directory does exist, there is a somewhat morecomplex operation, similar to a file delete. Specifically, a renamewhere the source and target directories are the same, and where thetarget of the rename exists (and will be unlinked), works similarly to afile delete operation, except that the locking is a bit more complex,since there are two names to lock in the parent directory, as well asthe target object's file handle. A locking hierarchy is chosen thatorders locks by file handle first, and then by file name within the filehandle (for locks that include a file name component). With this lockingorder, either both file names are locked in the directory before lockingthe target file, or both file names are locked in the directory afterlocking the target file. A rename in this case begins by creating a POT22 entry for the rename operation, and locking both the source andtarget file names within that directory. It then sends aperform-unlink-target operation to the target file's vdisk 24 (thesecondary vdisk 24), setting the flag saying that the target should waitfor the lock only if the target's file handle is ordered after thedirectory's file handle in the locking order. If the target succeeds atsetting the lock, it creates a POT 22 entry for the rename operation,and unlinks the target. It then responds to the primary vdisk 24, whichcompletes the rename operation by removing the source directory entryand changing the target directory entry to point to the source file'sfile handle. The message flow is the same as for the simple file deletecase illustrated in FIG. 6. If the attempt at locking the target fails,the rename operation then drops all of its locks, and sends aprepare-unlink-target to the secondary vdisk 24, and, upon receiving aresponse, then locks the parent directory's source and target filenames. At this point, it verifies that the target file's identity isunchanged; if it has changed, the entire operation restarts. Otherwise,the primary vdisk 24 sends a commit-unlink-target operation to thesecondary vdisk 24, while updating the directory as in the first case.Finally, the primary vdisk 24 sends a response back to the caller, andbatches a persistent-op-complete operation to clean up the state on thesecondary vdisk 24. FIG. 7 shows the message flow for this case, withthe difference that the directory update is as is described above inthis paragraph.

In the most complex rename case, when the source and target directoriesdiffer, the operation is significantly more complex and expensive. Thesource directory's vdisk 24 server begins by creating a request thatstarts by sending a lookup operation to the target directory's vdisk 24,looking up the target handle, while simultaneously locally looking upthe source file handle. Once these operations complete, the identity ofall objects involved in the rename operation is known, but no locks areheld.

The source vdisk 24 server then sends a prepare-rename-source message tothe source object's vdisk 24, locking that object; it sends aprepare-rename-target message to the target object's vdisk 24; and itsends a prepare-rename-tdir message to the target directory's vdisk 24.Each of these operations locks the entity in question, with the sourcedirectory's vdisk 24 locking the source directory, and sending thesemessages sequentially in the order required by the global lockinghierarchy. Once all of the entities are locked, the source vdiskverifies that the results of the initial lookups remain unchanged; ifnot, the entire operation restarts. Note that the prepare_rename_tdiroperation performs the target lookup verification step itself, to avoidrequiring an additional message exchange. Once all entities areprepared, locked and verified, the source then sendscommit-source-object, commit-target-object and commit-target-directoryoperations to each of the source-object vdisk 24, the target-objectvdisk 24, and the target-directory vdisks 24, respectively. Once thoseoperations have completed, the request enters the completed state at thesource directory vdisk 24, and batches up completion notifications toall of the other vdisks 24, so they can free their state knowing thatthe source directory vdisk 24 will never again send operationsassociated with the completed request.

FIG. 11 shows the message flow in the most general version of the renamevdisk 24 operation.

The persistent operations table is utilized in the functioning of theabove operations. Here is an example POT 22:

Primary Secondary Rename2 Rename2 UUID OpCode Dir FH FH/Name Dir FHFH/Name Status 1 Create VDisk = 1 VDisk = 2 NA NA Complete Inode = 100Inode = 121 Name = “foo” 2 Delete VDisk = 1 VDisk = 3 NA NA Remote Inode= 102 Inode = 122 Done Name = “bar” 5 Rename VDisk = 1 VDisk = 30 VDisk= 3 VDisk = 11 Remote Inode = 110 Inode = 130 Inode = 211 Inode = 212Sent Name = “m” Name = “p” 4 Perform VDisk = 30 VDisk = 1 NA NA ResponseUnlink Inode = NA Inode = 132 Sent Target Name = NA

In the table above, a set of POT 22 entries for vdisk 1 is shown. Forthe first four entries, vdisk 1 is the primary vdisk 24, driving thedistributed file system 10 operation in question. In the last entry,vdisk 1 is the secondary vdisk 24 for a delete operation being drivenfrom vdisk 30.

In more detail, the first POT 22 entry describes a file create operationfor a file name “foo” being created in the directory whose file handleis 1.100 (in the format <vdisk.inode>). The file is being created invdisk 2, selected by policy rule associated with vdisk 1. The operationhas run to completion, and when the response from the secondary vdiskwas received, it included the actual allocated inode from vdisk 2,specifically inode 121, which was entered into this POT 22 entry aswell. The operation is in “complete” state, meaning the operationcompleted at the secondary vdisk, and the primary vdisk has completedits work as well. The entry remains in the POT 22 only until the entryfor this same UUID in vdisk 2's POT 22 entry can be removed.

The second entry describes a file delete operation, where the directoryholding the file being deleted has file handle 1.102. The file beingdeleted has file handle 3.122, which was obtained by looking up the filename “bar” in the primary directory; from its file handle we see that itis stored in vdisk 3. The POT 22 entry is in “remote done” state,indicating that the secondary vdisk has freed the file, but the primaryvdisk 24 has yet completed the removal of the directory entry from theprimary directory itself.

The third entry (UUID 5) describes a rename operation, where the sourceand target directories have file handles 1.110 and 3.211, respectively.The file being renamed has file handle 30.130, which was determined bythe primary vdisk 24 doing a lookup operation on “m”, the file's name.The new name for the file is “p”, and that file already exists on vdisk11, with file handle 11.212. The operation is in “remote sent” state,meaning that primary vdisk is waiting for responses from the secondaryvdisks 24 before it can continue. Note that there are three remotevdisks 24, 3 (holding the second, target directory), 30 (holding thefile being renamed) and 11 (holding the file being deleted by virtue ofbeing the target of the rename operation).

Finally, the fourth entry describes a delete operation where thedirectory holding the file being deleted is stored on primary vdisk 30,which is driving the delete operation for a file that happens to bestored on vdisk 1 (the secondary vdisk for this operation); the primaryvdisk is requesting that this vdisk destroy that file, based on its filehandle. The vdisk of the parent directory is 30, and the secondary vdiskdoes not need to know the inode component of that file handle. The filebeing deleted has file handle 1.132, and once that file has beendestroyed, a response is sent back to the primary, telling it that theoperation requested by transaction ID 4 is complete. Note that the entryis in “response sent” mode, meaning that the operation is complete atthe secondary, and a response has been sent to the primary vdisk module.

The primary use of the persistent operations table (POT 22) is infailure recovery, where a processor 26 handling one of the vdisks 24involved in a multi-vdisk directory update fails. Essentially, a POT 22entry acts as an intentions log entry, describing the operation to beperformed, and whether that operation completed successfully or not.

In all of the directory modifying operations described above in thissection, there is a primary vdisk, which is the vdisk 24 at which theoperation begins execution. For any given operation, the POT 22 entriescreated by the primary vdisk 24 are called primary POT 22 entries.Similarly, the vdisks 24 contacted by the primary vdisk 24 to performpart of a directory operation are called secondary vdisks 24, and theirPOT 22 entries are called secondary POT 22 entries. Note that theselabels apply to the role a vdisk 24 and its POT 22 entries play in aparticular operation; thus, a vdisk 24 may be the primary vdisk 24 for aparticular operation even while it is a secondary vdisk 24 for anotherconcurrently executing operation. For a given operation, the primary POT22 entry and its secondary POT 22 entries are implicitly linked togetherby sharing the same unique transaction ID. In the example above, thefirst three entries are primary POT 22 entries, and the last entry is asecondary POT 22 entry.

Each POT 22 entry may be in one of two major states. It may be in the“executing” state, meaning that the request is currently executing, orit may be in the “complete” state, meaning that the request is finished,and has stored its results in the POT 22 entry.

If the processor 26 that created the primary POT 22 fails, then uponrestart, the processor 26 restarts the request, skipping the stages ithas already executed, but resending any subsidiary requests to thesecondary vdisks 24. All of these retransmitted requests are resent withthe original transaction ID, so that they can be matched with anyalready existing POT 22 entries, should the secondary vdisk 24 alreadyhave received the request before the primary's failure. If a secondaryPOT 22 entry indicates that the request is still in “executing” state,the secondary simply continues its execution. If there is no secondaryPOT 22 entry, indicating that the request has not yet executed, the newrequest begins execution. Finally, if the POT 22 entry is in “complete”state, the results of the operation, stored in the POT 22 entry, aresent back to the primary vdisk 24, without re-executing the secondaryrequest.

Similarly, if a processor 26 processing a secondary POT 22 entry fails,then after recovering, if the secondary POT 22 entry is not in“complete” state, the operation is re-executed, skipping any portionsthat were already executed, and sending the response back to the primaryvdisk 24 when the request completes. If the request is in “complete”state, the response is re-sent to the primary vdisk 24 withoutre-executing the request. If the primary vdisk 24 does not recognize therequest's transaction ID, this means that the primary vdisk 24 hadalready received an earlier response from the secondary vdisk 24,completed the primary request, and cleaned up its POT 22 entry. In thiscase, the secondary vdisk can delete its POT 22 entry.

A basic disk block allocation policy is implemented by storing somedescriptive tags with every segment in the system 10. These tags areimplemented as a bitmap of administrator-defined attributes. Forexample, an administrator might define a “low latency” attribute, a“write efficient RAID” attribute and an “archival” attribute forstorage, and apply the “low latency” attribute to segments comprised offlash storage, the “write efficient RAID” attribute to segments storedin RAID 1 storage, and the “archival” attribute to segments stored inRAID 5 storage. Other, orthogonal properties could also be defined, suchas “rotating media” for drive-based segments, or “slow” media for datarotating below 5400 RPM.

In this basic policy implementation, each Mode has policy descriptorsfor three separate classes of storage used by the Mode. One policydescriptor describes the meta data associated with this file: inparticular, the indirect blocks used for locating the file's datablocks. A second policy descriptor applies to the first N blocks of thefile, and the final policy descriptor describes the storage holding theremainder of the file. Each policy descriptor may be represented as apair of bitmaps, a mask bitmap and a value bitmap. A segment matches oneof these pairs if the segment's tag bitmap, ANDed with the policydescriptor's mask bitmap, matches the descriptor's value bitmap. Thefields stored in each Mode to represent these policies are shows in FIG.12. In this structure, the field “Initial part block count” gives thevalue of N above, while each of the remaining boxes hold the mask andvalue bitmaps describing the segments to be used for indirect blockallocation, initial part data allocation, and the segment to be used forthe rest of the file's data allocation. Thus, for example, when a datablock for an offset less than N needs to be allocated by a cleaner, oneof the vdisk's segments is chosen from the set of segments whosedescriptive tag matches the Mode's second policy descriptor.

When a new file or directory is created, the policy descriptor shown inFIG. 12 is inherited from the object's parent directory, and used toselect storage for the data written to the newly created object.

There are a number of operations used for managing policies and thefiles using policies.

-   -   GetPolicy—returns the policy object associated with a file or        directory.    -   SetPolicy—sets the policy object associated with a file or        directory.    -   Recursive apply—sets the policy object for a directory, and all        of its descendent directories and files.

Note that this is an exemplary policy description, and that many othersare possible, including policy descriptors that are only applied if afile's attributes (size, owner, etc) have certain values, or the file'sname matches a regular expression. One might, for example, define apolicy that says all files whose name match the regular expression“.*\.o” and whose size is greater than 1 MB should be stored on“archival” class storage.

When the policy description in an Mode is updated, the current datalayout for the corresponding file may no longer match the updatedpolicy. In this case, the inode is placed in a persistent queue for abackground process to update the Mode data's allocation to match the newpolicy. Similarly, when new storage is placed into service for a vdisk24, the existing policy descriptors may, upon re-examination, selectsome of the new storage to hold meta-data or normal data. In this case,all of the Modes need to be placed in a queue for a background task toverify that the inodes' block allocations still match the allocationpolicies.

For example, the administrator might add some new, very low latencystorage to a vdisk's segment list, and then specify a new policy for oneor more inodes to place the first 64 KB of data in that new segment. Thesystem 10 would do this by performing an “AddSegment” operation to addthe storage to the vdisk 24, adjusting the table describing whichsegments are part of the storage space for a vdisk 24 to include the newsegment, possibly followed by a SetPolicy operation to specify where touse this new storage for a portion of the file system name space. Thesystem 10 would then internally perform a “readjust” operation on theinode or inodes in question, as the policies are updated, which wouldcheck that each block in each file whose policy is updated is allocatedfrom a segment with the appropriate class of service. If a file failsthis test, then the readjust operation would, for every block allocatedfrom the wrong segment, allocate a block in the desired segment, copythe data from the old segment to the new segment, and adjust theindirect block pointing to the original block to point to the newblock's location.

The preceding described the operation of the system 10 in terms ofoperations on individual Modes. These Modes are implemented in a mannersimilar to that in which Unix systems like FreeBSD implement inodes intheir UFS file system, with the exception that in this invention, apolicy-based block allocator is invoked to choose the segments fromwhich to obtain blocks to add to a file. This allocator is invoked anytime that any of the operations discussed in this section need toallocate more disk blocks to an existing or new file, directory, orsymbolic link.

It is assumed that when a file or directory is created, it inherits areference to a policy object from its parent directory. This policyspecifies in some manner which blocks within the file should beallocated with which CoS from among the segments making up the file'svdisk 24. For example, one such policy might be the disk block allocatordescribed above.

When disk blocks are later allocated to such a file, the file's policyis consulted, and this policy specifies the desired class of service forthe newly allocated blocks. The block allocator then chooses a segmentwith the desired CoS from among those within the file's vdisk 24, andallocates space from that segment's block number subspace. Note thatsince all of the segments in a vdisk 24 reside at different locationswithin the same block number space, once blocks have been chosen to holda file's newly added data blocks, the rest of the file allocationprocess proceeds in the same way as allocating space to files in theFreeBSD system.

The blocks within a segment are located via a very simple per-segmenttable that identifies the block number space for each segment withineach vdisk 24. For example, the segment table shown below might specifythat the dotted segment resides at vdisk 24 block offset 10000 for 5000blocks, and that the white segment resides at vdisk 24 block offset20000 for 3000 blocks. Once a segment has been chosen for blockallocation, the file system 10 determines the corresponding block numberrange for the segment, and then consults the UFS-like block allocationbitmap to determine which blocks are actually available in that segment.These block numbers are then stored in the UFS-like inodes and indirectblocks, just like the file system 10 stores any block number generatedthrough any other block allocation algorithm used by the file system 10.A global (per vdisk 24) policy describes the class of service desiredfor the allocation of global file system meta data, such as inodes;typically, these would be allocated from relatively low latency storage.

The internals of a system 10 designed to implement a file system 10comprised of multiple vdisks 24 is based upon implementing a set ofvnode interface 18 calls on top of the vdisk 24 layer. Most vdisk 24operations have analogous vnode operations, so that the interface 18function for those operations is very simple.

FIG. 13 shows the relationship between vnodes, vdisks 24, and segments,in the context of a Unix operating system such as OpenSolaris running afile system 10 based on this invention. In FIG. 13, there is a VFSinstance for each exported file system tree, and each such instance iscomposed of one or more vdisks 24.

In FIG. 13, it is shown how the vdisk 24 manager can fit into a standardUnix kernel. The figure shows three file systems 10 (and thus three namespaces). The first, VFS A, is comprised of files from vdisk 1 and vdisk2. The second, VFS B, is comprised of files allocated from vdisk 3. Thethird, “Local UFS VFS” is a file system implemented from a normal Unixfile system on its own local disks 20.

The VFS shim layer implements a very thin layer mapping vnode operationsto vdisk operations, and is described below in detail.

Once a vdisk 24 operation is invoked, some vdisk 24 operations,especially the directory modification operations described above,require performing internal vdisk 24 operations at other vdisks 24implementing the same VFS. For example, a file create performed on adirectory within VFS A that happens to be located on vdisk 1 mightcreate its file on vdisk 2, and thus might invoke theperform-create-target function on vdisk 2. The block labeled “VDisk <n>Secondary” represents the server for those internal vdisk operations,and the dashed arrows indicate that these internal vdisk operations aretypically invoked by the primary vdisk implementation for some othervdisk within the same file system tree (and thus the same VFS).

Each vdisk 24 implements its own pool of inodes, and so needs to storedata persistently. Each vdisk 24 has a set of segments, accessed via theinterface 18 described below, that it uses to store its persistent data.Different segments provide differing classes of service. For example,vdisk 1 has reasonably fast disk storage in a RAID 1 segment, and someextremely fast but expensive storage in a flash memory 12 segment. Thevdisk 24 may have, for example, an automatic allocation policyspecifying that the first 256 KB of each file should be allocated fromblocks in the flash segment, and any additional storage should beallocated from the slower disk segment. If this policy is followed, forexample, sequential access to a randomly chosen set of files would bevery fast, since the first 256 KB of data would be available nearlyinstantaneously, during which time the disk arms are positioned totransfer the remainder of the file.

Similarly, vdisks 2 and 3 are compromised of a mix of relativelyinexpensive RAID 5 storage, along with more expensive, but betterperforming RAID 1 storage.

The vdisk interface 18 consists of two sets of functions, a primaryinterface 18 called from the vnode layer on incoming file system calls,and a secondary interface 18 invoked by directory modifying primaryvdisk operations that span multiple vdisks 24.

The calls in the vdisk 24 primary interface 18 are typically called fromthe vnode layer. Many of the calls are applied to, or take, VDiskInodeparameters describing the files themselves. The following describes theoperations in the vdisk primary interface used by the vnode layer; somecommon operating system specific parameters, such as authenticationcredentials, have been omitted for clarity:

VDiskInode::release( )—decrease the reference count on an inode.

VDiskIndode::hold( )—increment the reference count on an inode.

VDiskInode::getattr(VDiskInodeAttr *attrsp)—get attributes associatedwith an inode.

VDiskInode::setattr(VDiskInodeSetAttr *newAttrsp, VDiskInodeAttr*updatedAttrsp)—update attributes as described by newAttrsp, returningthe updated attributed in *updatedAttrsp.

VDiskInode::blocckRead(uint64_t offset, uint32_t count, uint32_t flags,uint32_t *bufCount, buf **bufpp, VDiskInodeAttr *attrsp)—read data fromMode starting at offset, for count bytes. The value *bufCount on inputspecifies the size of the bufpp array, and on output is set to theactual number of referenced buffers returned. The Mode's attributes atthe time of the read are returned in *attrsp. Note that a flag of 1specifies that buffers should be obtained held for writing instead ofreading, indicating that the operation is part of a write operation thatwill modify the buffers.

VDiskInode::truncate(uint64_t offset, uint32_t count, VDiskInodeSetAttr*newAttrsp)—zero bytes from byte ‘offset’ for ‘count’ bytes. Any wholeblocks that can be freed are freed. The newAttrsp parameter optionallymay specify updated mtime, ctime or atime values for the file.

VDiskInode::readdir(uint64_t *cookiep, uint32_t *countp, char *resultsp,uint32_t *flagsp)—returns directory listing results, starting at anopaque value. In the initial call to readdir, *cookiep should be 0.*countp gives the number of bytes available in the *resultsp buffer.Each entry consists of a 16 bit file name length, followed by a UTF-8encoded file name, followed by a 64 bit “Mode number”, followed by a 64bit cookie value for the directory entry following this entry. Anintegral number of directory entries is always returned by readdir. The1 bit is set in *flagsp if EOF is encountered, and no further entrieswill be returned. Note that *countp is updated to indicate the actualnumber of bytes returned, and *cookiep is updated to give the cookievalue that, when passed into a new instance of this call, will returnthe next entry in the directory listing after those returned by thiscall.

VDiskInode::lookup(char *namep, VDiskInode **newInodepp)—lookup the name‘namep’ in the specified directory, returning a held reference to thetarget Mode in *newInodepp. If the entry can't be located, a non-zeroerror code is returned. A return code of ENOENT is reserved for anindication that no temporary error occurred, and the file namedefinitely does not exist in the directory.

VDiskInode::readdirplus(uint64_t *cookiep, uint32_t *countp, uint32_t*entriesp, char *resultsp, VDiskInodeAttr *attrsp, uint32_t*flagsp)—This function acts like a combination of a readdir operationfollowed by getattr operations for each file entry returned. Theparameters are the same as in readdir, with the following changes. Thefield *entriesp on entry gives the space available to hold returnedattributes in the attrsp array, which points to an array of attributestructures. On exit, this field is set to the number of entries actuallyreturned. The information returned in attrsp is ordered the same as theentries in the resultsp array.

VDiskInode::create(char *namep, VDisk *newVDiskp, VDiskInodeSetAttr*newAttrsp, VDiskInode **inodepp, VDiskInodeAttr *attrsp)—create a newfile in the specified directory, using the new attributes specified bynewAttrsp. The resulting attributes are returned in *attrsp, and a newinode is returned in *inodepp. Note that the file is created in a newvdisk specified by the newVDiskp parameter; if this parameter is null,the target vdisk is determined by consulting the policy choice moduleshown in FIG. 5.

VDiskInode::remove(char *namep, VDiskInode **inodepp, VDiskInodeAttr*attrsp)—remove a file or symbolic link from the specified directory.The updated object attributes are returned in *attrsp; this ismeaningful for file's whose link count was decremented, but not to zero;in this case, *inodepp will also be set to a held reference to the inodein question.

VDiskInode::mkdir(char *namep, VDisk *newVDiskp, VDiskInodeSetAttr*newAttrsp, VDiskInode **inodepp, VDiskInodeAttr *attrsp)—create adirectory with the specified name, with the new attributes specified by*newAttrsp. A held reference to the newly created inode is returned in*inodepp, and the newly created file's attributes are returned in*attrsp. Note that the file is created in a new vdisk specified by thenewVDiskp parameter; if this parameter is null, the target vdisk isdetermined by consulting the policy choice module shown in FIG. 5

VDiskInode::rmdir(char *namep)—remove the directory named ‘namep’ fromthe directory to which this operation is applied.

VDiskInode::symlink(char *namep, VDisk *newVDiskp, char *contentsp,VDiskInodeSetAttr *newAttrsp, VDiskInodeAttr *attrsp)—create a symboliclink named ‘namep’ with initial contents ‘contentsp’. The initialattributes are set from *newAttrsp, and the resulting full attributesare returned in *attrsp. Note that the file is created in a new vdiskspecified by the newVDiskp parameter; if this parameter is null, thetarget vdisk is determined by consulting the policy choice module shownin FIG. 5.

VDiskInode::link(char *namep, VDiskInode *inodep, VDiskInodeAttr*attrsp)—create a hard link with name ‘namep’ in the specifieddirectory, to the object specified by the Mode *inodep. Updatedattributes for the target object are returned in *attrsp, as ofimmediately after the operation's completion.

VDiskInode::rename(char *snamep, VDiskInode *targetp, char *tnamep,VDiskInode **sinodep, VDiskInode **tinodep, VDiskInodeAttr *sattrp,VDiskInodeAttr *tattrp)—rename the file name ‘snamep’ in the source(applied) inode, changing its name to ‘tnamep’ in the target directory‘targetp’. A read reference to the updated source Mode is returned in**sinodep, and a reference to the target Mode is returned in *tinodeppif the target object continues to exist after the unlink operation.Attributes immediately after the operation are returned in *sattrp and*tattrp for the source and target objects, respectively.

The operations above refer to the following non-opaque structures: bufs(disk buffers), VDiskInodeAttr and VDiskInodeSetAttr structures. Thefollowing tables show the contents of each of these structures:

The buf structure represents a disk buffer. While there are manypossible implementations of such a structure, the key fields that aretypically present include the virtual and physical addresses of the datain the buffer, as well as a reference count, that, when zero, indicatesthat no processes are actively accessing the buffer.

buf structure Data char * Points to mapped in data buffer. PhysAddrchar * Points to phyocation of buffer. RefCount uint32_t Reference countfor buffer.

Next, the VDiskInodeAttr structure is described; this gives informationsimilar to the Unix “struct stat” structure:

VDiskInodeAttr structure Device uint64_t A unique value indicating thevdisk storing the file. Inode uint64_t A unique value indicating theinode with the vdisk/device corresponding to this file. UnixModeBitsuint16_t 16 bits of file mode as defined by POSIX file systemspecification (1003.1). This includes both the file type as well as thebasic file protection bits. Owner uint32_t File owner's ID. Groupuint32_t File's group owner. Length uint64_t File length in bytes.Access time 2 × Time file data last accessed, in seconds uint32_t andmicroseconds since midnight, 1/1/1970 GMT. Modification time 2 × Timefile data last modified, in same uint32_t format. Change time 2 × Timefile attributes last changed, in same uint32_t format. Space uint64_tSpace allocated for file, in bytes.

The VDiskInodeSetAttr structure includes all of the fields of aVDiskInodeAttr structure, plus a single bit for each field, which is setto 1 if the value is valid in the structure, and should be set into thefile's attributes, and 0 if the value should be ignored. This structureis passed to calls that set or change file attributes, and only thefields for which the corresponding bit is set are updated by the call.

Note that in all cases except for persistentOpComplete, the POT 22 entryremains in the persistent operations table until a persistentOpCompleteis received for the entry. At any time, the secondary may also verifythat the operation tagged with a POT 22 entry's request ID is stillpresent at the primary; if it isn't, this means that the operationcompleted, the persistentOpComplete request got lost, and the secondarycan and should remove the operation's entry from the POT 22 as well.

VDisk::performUnlinkTarget(UUID requestId, VDiskInode *targetp)—CreatePOT 22 entry with ID requested, then lock the file handle for Modetargetp, decrement the target object's link count, and drop the filehandle lock, leaving the POT 22 entry to catch duplicate operations.

VDisk::prepareUnlinkTarget(UUID requestId, VDiskInode *targetp)—CreatePOT 22 entry with ID requested, and lock the file handle for Modetargetp, leaving locks set and POT 22 entry to catch duplicateoperations.

VDisk::commitUnlinkTarget(UUID requestId)—Unlink the locked objectreferenced from the exiting POT 22 entry, drop the lock and return. Thiscall expects to be applied to a requestId specifying an operation thathas already performed a prepareUnlinkTarget operation, which specifiedthe Mode to be unlinked.

VDisk::performLinkTarget(UUID requestId, VDiskInode *targetp)—Create aPOT 22 entry with the specified requestId, lock the specified object,increment object's link count, drop locks and return. The entry remainsin the POT 22 until the receipt of a persistentOpComplete request withthe same transaction UUID.

VDisk::prepareLinkTarget(UUID requestId, VDiskInode *targetp)—Create aPOT 22 entry with ID from requestId, lock the specified object *targetpand return.

VDisk::commitLinkTarget(UUID requestId)—Increment the link count on theobject locked by the request and stored in the POT 22 entry, update itsattributes, drop the lock on the target and return. This call expectedto be performed on a POT 22 entry for which a prepareLinkTargetoperation has already been successfully performed.

VDisk::performCreateTarget(UUID requestId, VDiskInode **targetpp,VDiskInodeSetAttrs setAttrs)—Create a POT 22 entry for the request ID,allocate the target file Mode (storing it in the POT 22 entry) andreturn the new object's file handle to the caller. On a retransmissionof this request, resend the already allocated object's file handle. ThesetAttrs parameter specifies the type of object to be created, as wellas its initial attributes.

VDiskInode::lookup—This is the same function as present in the primaryinterface 18; it can also be invoked via the secondary interface 18.

VDisk::prepareRenameOperand(UUID requestId, VDiskInode *targetp)—CreatePOT 22 entry, lock file handle of target Mode, and return. Thisoperation is performed for the target directory Mode, the source objectMode and the target object Mode for rename operations affecting twodirectories. For single directory renames, the operation is performed onthe source object Mode, and, if it exists, the target object Mode.

VDisk::commitRenameTargetDir(UUID requestId, char *namep, VDiskInode*inodep)—Update the target directory's entry for the target ‘namep’ topoint to the file handle for the mode specified by inodep. Then drop alllocks on inodep's file handle, and return. This call requires that aprepareRenameOperand call have been previously made with this requestIdto this server.

VDisk::commitRenameTargetFile(UUID requestId)—Update the target inodeassociated with the request's POT 22 entry by decrementing its linkcount, then drop its locks and return. This call requires that aprepareRenameOperand call have been previously made with this requestIdto this server.

VDisk::commitRenameSourceFile(UUID requestId, VDiskInode *targetDirp)—Ifwe're renaming a directory, update the “..” pointer in the directoryassociated with the locked file handle (obtained via the POT 22 entry),drop the locks and return. This call requires that aprepareRenameOperand call have been previously made with this requestIdto this server.

VDisk::persistentOpComplete(UUID requestId)—This operation removes thePOT 22 entry tagged with requestId from the persistent operations table.To guard against this message being lost due to a poorly timed systemcrash, the secondary also periodically verifies that old POT 22 entriesare still valid by contacting the primary vdisk with theVDisk::checkOperationStatus call to determine whether the POT 22 entry'srequest ID is still active.

VDisk::checkOperationStatus(UUID requestId, int *statusp)—This operationchecks on the status of the transaction tagged with requestId. The valueof *statusp is set, on return, to the status of the operation, that is,one of remoteSent, remoteDone, complete, or unknown. The first three arenormal states recorded in a POT 22 entry, while the last indicates thatthere is no POT 22 entry with the specified transaction UUID.

Every operation in the vnode interface 18 has a corresponding operationin the vdisk 24 interface 18, having the same name. Most operations inthe vdisk 24 interface 18 require a subset of the input parameters ofthe corresponding vnode operation with the same name, or return asuperset of the output parameters required by the vnode operation, andso can trivially be implemented by calling the corresponding vdisk 24operation with the underling VDiskInode object (which can even beembedded in the vnode structure used by the vnode interface 18).

The segment interface 18 provides operations for reading and writingfile data, and reading and caching directory information. The followingoperations are provided:

Segment::read(uint64_t offset, uint32_t count, buf *datap). This callreads the data located at the specified offset, returning it in theprovided buffer. Only count bytes are transferred.

Segment:write(uint64_t offset, uint32 t count, buf *datap). This callworks like read, only the data is written from the start of the providedbuffer.

Segment::readdir(uint64_t offset, uin32_t count, buf *datap). This callreturns a block of directory entries in a standard form from a physicaloffset within a directory. The data returned is an integral number ofrecords, each giving a file name (including the file name's length inbytes), and the 64 bit Mode number within the vdisk 24 of the file.

Note that these calls can be executed as local calls to access drivesconnected to the same computer system as the caller, or as remoteprocedure calls to access drives connected to other computer systems.

This invention provides two significant advantages over the state of theart today in file systems. First, the invention provides a much moreflexible mechanism for changing the class of service of files, and evenportions of files, than traditional file systems, both with directedattached storage and network attached storage systems. By class ofservice, we mean without restriction any property of the storage such astransfer rate, request latency, reliability or expense. Second, theinvention provides a significantly simpler administrative model for filesystems that can be serviced by multiple processes, for example, on amultiprocessor or on multiple systems in a clustered server. Because theinvention divides a file system into an arbitrarily large number ofindependently servable and individually repairable componentsautomatically, rather than by requiring the administrator to define aset of volumes, the administrative model is much simpler, and theresulting system is much easier to service.

For example, with this invention, an administrator could specify apolicy where the first megabyte of every file would be located onsegments having very low latency, perhaps comprised of very fast disks20 or flash memory 12. The remaining blocks would be allocated fromnormal storage vdisks 24. With an operations mix that chooses files atrandom and then reads each chosen file sequentially, this policy wouldreduce overall latency to the data in these files, since the firstportion of the data would be accessible at very low latency, and duringthe transfer of this first portion of the data, the remaining data couldbe accessed from drives with a higher latency.

In many of today's storage systems, there are several options forchanging the class of service of stored data. Systems like IBM's AFS,the Open Software Foundation's DCE/DFS, NetApp's Ontap GX, and Sun's ZFSprovide a mechanism for moving volumes, representing subtrees of thefile system name space, from one storage area to another. When the twostorage areas provide different classes of storage, the administratoreffectively changes the class of storage for the relevant subtree whenmoving the volume from one area to another. This invention improves uponthis art in several ways. First, volume boundaries are administrativelydifficult to adjust after the creation of the volumes, while thisinvention does not have a comparable volume concept whose boundariesmight need adjustment to match the desired class of service boundaries.Instead, this invention provides multiple classes of storage within theblock address space used by a single pool of inodes (a vdisk 24), sothat any file can be moved to storage with a new class of storage at anytime, without changing where it resides in the file system name space.Second, class of service policies that adjust the class of service fordata stored in a file system, in this invention, can make adjustments ona block-by-block basis. The above systems would all need to relocate anentire volume to make any class of service adjustments, and wouldfurthermore be unable to make any adjustments at any granularity belowthat of an entire directory and its contents, while this invention canadjust data's class of service on a file by file, or even a block byblock basis.

In terms of administrative model simplicity, again, comparing thisinvention with volume-based data architectures, this invention has theadvantage that no volume boundaries have to be chosen at all—instead,data is randomly distributed among vdisks 24. The class of service ofthe storage is associated not with the specific vdisk chosen, but withthe type of segment storing the data within a particular vdisk 24.

In the realm of serviceability, instead of having to run diskconsistency checks over the entire file system, or over a single volume,both administratively visible concepts, in the invention, diskconsistency checks are run over individual vdisks 24. Vdisks 24 are notindividually managed by administrators, so that having many vdisks 24making up an individual file system does not add administrativecomplexity to the system management task.

In terms of meta data scalability, this invention improves on the stateof the art for a global name space in a number of ways. As compared witha name space with a meta data synchronization server, such as Red Hat's(originally Sistina's) GFS, this system performs indirect block updatescompletely within an individual vdisk 24, without any communication withother vdisks 24. This system also performs directory updates on at mosttwo vdisks 24 (except for the infrequently executed rename operation,which typically involves one vdisk 24, but can in some complex casesinvolve up to four). When multiple vdisks 24 collaborate on a singledirectory operation, they do so by exchanging a small number of messagesamong themselves (as described above), where each individual operationactually executes on a single vdisk 24. Since operations on each vdisk24 can be performed by separate processors 26 without any references todata structures controlled by other vdisks 24, this architecture allowssignificant global file system scaling without requiring a meta dataserver acting as a synchronization point, via the splitting of an activefile system into a moderately large number of vdisks 24. Thus, becausefiles are distributed among vdisks 24 automatically, a single name spacecan be distributed among multiple processors 26 without any manualadministrative intervention.

As compared with systems like IBM's AFS, the OSF's DCE/DFS and NetApp'sOntap/GX, which divide the global file system into a number ofindependent subtrees (volumes), this invention's scalability benefitscome from its ability to divide the files within a given directory intoa number of vdisks 24, all of which can be processed independently (asopposed to the above systems, which require that all files within asingle directory reside in a single volume, and thus be served by asingle processor 26 system.) In addition, because vdisk 24 creation anddeletion can be automated much more easily than volume creation anddeletion (since the latter requires an administrator's choosing thevolume boundaries in the global name space), this invention allows forthe creation of many more vdisks 24 than the above systems, allowingautomatic load balancing algorithms more flexibility to smoothlydistribute vdisks 24 across processors.

In terms of class of service management (CoS), this invention isbelieved to improve on the state of the art in a number of ways. Thestate of the art in CoS management has been to relocate volumes indesigns such as AFS, DCE/DFS and Ontap/GX from underlying storage withone class of service to underlying storage having a different class ofservice. The weaknesses of this approach are that the granularity of thedata whose CoS is changed is that of an entire volume, that all of thedata within the volume are copied in order to change the CoS of any ofthe data, and that the volume boundaries are chosen initially to matchthe boundaries at which the administrator, sometime in the future, willrequire for CoS updates. Changing volume boundaries after volumecreation is both complicated and difficult to do while the data is beingaccessed by clients concurrently, since file handles held by clientsystems include a volume ID as part of that file handle. This means thatoperations changing volume boundaries will change client resident filehandles, limiting the transparency of those operations. This invention,on the other hand, determines the CoS for an individual block of a fileby choosing the appropriate segment from which to allocate the block,and thus operates at a lower level of abstraction than file handles. Noadministrative boundaries need to be determined or changed beforechanging a CoS policy. Furthermore, if new data with a different classof service becomes available, it can be divided into a number ofsegments, and each segment can be joined to existing vdisks 24automatically, providing convenient access to the new type of storagefor existing files. A new policy could then specify which blocks ofwhich files should use the new storage, and that new storage could beautomatically used for newly written files, while in the background,files conforming to the new policy could have their data migrated intothe new. In this case, in other words, the ability to dynamically addnew segments to a vdisk's block address space, combined with invention'sability to allocate and reallocate file data from any of a vdisk'ssegments, allows a very inexpensive CoS management mechanism that canspecify different classes of service at a very fine level of granularity(that of individual blocks in a file), and that can also change theclass of service of an existing file continuously in the background,also on a block by block basis.

A glossary of various terms used here follows.

-   -   Chunk—A fixed sized, contiguous portion of a single disk. Chunks        may store data or checksum/parity information. Multiple chunks        sharing the same class of service, or basic attributes, are        concatenated into segments, a variable sized piece of storage.    -   File attributes—Meta data information describing the properties        of a file, including the file's length in bytes, the user ID of        the owner of the file, the file's last accessed time, last        modified time and last “attributes modified” time.    -   Persistent Operations Table (POT 22)—A per-vdisk table tracking        the progress of atomic file system operations that affect a        single VFS, but one or more vdisks 24. For example, a file        create or delete may affect a directory stored within one vdisk        24, and a file stored within another vdisk 24 in the same        virtual file system. The persistent operations table on each        vdisk 24 keeps track of the progress of each such file system        operation. All entries describing the progress of a single        atomic operation are tagged with the same operation UUID.    -   RAID array—An array of physical disks grouped together with some        form of RAID parity scheme, and storing a number of fixed sized        chunks.    -   Segment—A variable length collection of a number of chunks, all        sharing the same type of storage, for example, RAID 1 storage        comprised of 15K RPM disk drives. A segment can be addressed        internally by a virtual 64 bit block pointer; these addresses        only map to the data chunks of a segment, not the parity chunks.        The virtual addresses are all contiguous within a single        segment, but the underlying physical addresses of the individual        chunks of which the segment is made may be scattered throughout        the disks 20 attached to a computing system.    -   Segment Interface—A simple interface providing operations to        read and write data stored in a segment.    -   UUID—Universal Unique IDentifier, a 128 bit, easy to construct        identifier that is unique over all systems and all time.        Typically, these are constructed using the IEEE 48 bit hardware        address of some card in the computing system, combined with a        very fine granularity clock value, and a process ID and/or boot        counter.    -   VDisk—An arbitrary collection of inodes, not connected as a        single file system tree, storing its data and meta data in a        dedicated collection of segments. The different segments within        a vdisk 24 may have different properties, e.g. RAID levels,        transfer rates, etc, and individual files may be allocated        entirely from one segment within a vdisk 24, or from multiple        segments, depending upon externally provided policies. For        example, one possible policy might be to place all meta data in        a RAID 1 vdisk, and put all user data in a RAID 5 vdisk. Another        possible policy might be to store the first megabyte of every        file in a segment having very low latency (perhaps comprising        flash memory 12), with the remaining blocks allocated from        normal RAID 5 storage, so that reading random medium-sized files        in their entirety could be done with very low overall latency.        One or more vdisks are combined to create a VFS, or file system.    -   VDisk Interface—An interface used in this invention to perform        file system modifying operations on files stored in vdisks 24        making up a single VFS. The key new functionality in the vdisk        interface allows directory operations to change objects in more        than one vdisk, by beginning execution at one of the vdisks 24,        which then forwards subsidiary requests to the other vdisk(s)        involved in the operation on a secondary interface 18.    -   VDisk Primary Interface—The primary interface used by components        such as the local NFS server to access files stored in the        vdisks 24 making up a VFS. For each VFS operation, there is a        corresponding vdisk operation, with slightly different        parameters, as described in the section above on the VDisk        interface.    -   VDisk Secondary Interface—The interface invoked by those vdisk        primary interface operations that update objects on more than        one vdisk, to effect changes to those objects on the other        vdisks 24. For example, the secondary vdisk interface includes        an operation to allocate an inode, which is invoked by the        primary vdisk file create operation. Most operations in the        secondary interface create POT 22 entries to ensure that their        changes occur atomically with respect to the invoking primary        interface operation.    -   VFS or Virtual File System. A collection of files and        directories stored in one or more vdisks 24, and making up        together a connected file system tree, with a root directory and        a collection of subdirectories, each containing other files and        subdirectories. A VFS contains a number of vdisks 24, and each        vdisk is a member of exactly one VFS.    -   VFS Interface—A reasonably standard interface to virtual file        systems, first introduced in the 1980s by Sun Microsystems in        the SunOS 3.X operating system, and today present in some form        in many Unix and Linux-based kernels, including Sun's        OpenSolaris operating system. Typical operations including        reading and writing blocks within files, reading and changing        file attributes, and creating and deleting files within a        directory.

Although the invention has been described in detail in the foregoingembodiments for the purpose of illustration, it is to be understood thatsuch detail is solely for that purpose and that variations can be madetherein by those skilled in the art without departing from the spiritand scope of the invention except as it may be described by thefollowing claims.

1. A storage system comprising: a computer; a plurality of vdisks incommunication with the computer, each vdisk containing a plurality ofpersistent storage segments, each segment providing a specific class ofservice (CoS) for storage different from the CoS for storage of theother segments; each vdisk storing one or more files with data and metadata distributed among its storage segments, a vdisk to hold a newlycreated file or directory is chosen from the plurality of vdisks basedon a predetermined mapping into the plurality of vdisks; and a policymodule for vdisk and segment choice in communication with the computer.2. A storage system as described in claim 1, wherein a first portion ofa file's data is stored in a first segment, and a second portion of thefile is stored in either the first segment or a second segment.
 3. Astorage system as described in claim 2, where the predetermined mappingis a round robin assignment algorithm.
 4. A storage system as describedin claim 2, where the predetermined mapping chooses the vdisk with alargest available space, or a largest percentage of available space. 5.A storage system as described in claim 2, including a plurality ofprocessors and wherein the predetermined mapping chooses the vdiskserved by a least loaded processor.
 6. A storage system as described inclaim 2, wherein each vdisk contains an arbitrary collection ofdirectories and files, and at least two vdisks hold at least one file.7. A storage system as described in claim 2, wherein each vdisk containsmeta data disk block pointers only to blocks within a same vdisk.
 8. Astorage system as described in claim 7, including an interface toinitiate a file system consistency check on an individual vdisk,triggered by an indication of an inconsistency in a specific vdisk.
 9. Astorage system as described in claim 1, wherein at least one file's datais stored in one segment, and at least one other file's data is storedin a second segment.
 10. A storage system as described in claim 9, wherethe predetermined mapping is a round robin assignment algorithm.
 11. Astorage system as described in claim 9, where the predetermined mappingchooses the vdisk with a largest available space, or a largestpercentage of available space.
 12. A storage system as described inclaim 9, including a plurality of processors and wherein thepredetermined mapping chooses the vdisk served by a least loadedprocessor.
 13. A storage system as described in claim 2, wherein eachvdisk is an arbitrary collection of directories and files, and at leastone file is stored in at least two segments.
 14. A storage system asdescribed in claim 9, wherein each vdisk contains an arbitrarycollection of directories and files, and at least two vdisks hold atleast one file.
 15. A storage system as described in claim 9, whereineach vdisk contains meta data disk block pointers only to blocks withinthe same vdisk.
 16. A storage system as described in claim 15, includingan interface to initiate a file system consistency check on anindividual vdisk, triggered by an indication of an inconsistency in aspecific vdisk.
 17. A system as described in claim 2 wherein eachsegment consists of a plurality of chunks.
 18. A method for storing afile comprising the steps of: receiving the file at a communicationinterface; storing a first portion of the file in a first segment of apersistent memory in communication with the interface and a secondportion of the file in a second segment of the memory having a differentclass of service for storage than the first segment's class of service;and retrieving by a computer the file by reading the first portion andthe second portion from the first and second segment, respectively, avdisk to hold a newly created file or directory is chosen from theplurality of vdisks based on a predetermined mapping into the pluralityof vdisks.
 19. A storage system comprising: a computer; a plurality ofvdisks in communication with the computer, each vdisk containing aplurality of persistent storage segments, each segment providing aspecific class of service (CoS) for storage different from the CoS forstorage of the other segments; each vdisk storing one or more files withdata and meta data distributed among its storage segments and apersistent operations table that implements directory modifyingoperations atomically, each vdisk is an arbitrary collection ofdirectories and files, and at least one file is stored in at least twosegments.
 20. A storage system comprising: a plurality of vdisks, eachvdisk containing a plurality of storage segments, each segment providinga specific class of service (CoS) for storage different from the CoS forstorage of the other segments; each vdisk storing one or more files withdata and meta data distributed among its storage segments, each segmenthaving disk blocks, the disk blocks being allocated to files and beingmarked as in use by a corresponding bit being set in the bitmapallocation table, indexed by physical block address; and a persistentoperations table which tracks file system operations that affect asingle vdisk.